fix(snapshot): authenticate exact incremental state - #17
Merged
farhan-syah merged 2 commits intoJul 26, 2026
Merged
Conversation
Incremental apply distinguished the base's reader-visible page set from its published set only for the write guard, then reused the wider published set to define which pages the delta must contain. That made a producer's legitimate reuse of a page still held by the follower's own free-list chain or commit-history tree look like a malformed artifact instead of the true base/target mismatch it is. Split the two page sets explicitly, report the real collision as a new SnapshotBasePageReused error, restrict segment restore to segment-id filenames, and record destination ownership so failed export/restore/ apply cleanup never deletes a directory the caller pre-created.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes full snapshot restore and incremental snapshot application fail closed
against the exact authenticated state their manifest describes, and adds the
regression coverage needed to tell a complete, durable apply from a
superficially successful copy.
The previous implementation validated the manifest envelope but not the
relationships between the manifest, main-file pages, segment files, the target
header, and the follower's existing state. That left room for malformed or
incomplete artifacts to be accepted, for undeclared state to survive a restore,
and for a retry to authenticate stale plaintext cached before a failed apply.
Why these checks belong together
A snapshot is only valid if all of the following describe the same state:
Validating any one of those in isolation is insufficient. A target header can be
structurally valid while naming a missing main page; a segment file can exist
under the expected name while carrying a different authenticated identity; and a
raw page rewrite can be correct on disk while a warm pager still holds plaintext
from the state that preceded it. Snapshot creation, staging, authentication,
publication, and cleanup are therefore treated as one ordered protocol.
The delta's page set is defined once
The base state contributes two page sets, and they are not interchangeable:
both sides of the protocol can compute, so it is what defines which pages a
delta carries. Producer and follower now derive it with the same formula:
target_reachable − base_reader_visible.apply_incrementalcarries both over from the follower's own header ratherthan from the producer, so those pages stay live across the apply and must
never be overwritten. The producer cannot see them, so they can only ever be a
write guard — never part of the delta's definition.
Conflating the two makes a healthy snapshot look malformed: subtracting
publishedwhen deciding which pages a delta should contain turns every pagethe producer legitimately allocated over a follower-local free-list page into an
unexplained set mismatch.
Page reuse is not a defect
There is deliberately no producer-side page-reuse check. A page id below the
base allocation cursor proves nothing: a page that was free at the base commit
is legitimately reallocated for the target, and shipping it is safe precisely
because nothing base-reachable points at it. The cursor is an allocation
watermark, not a liveness boundary — rejecting on it fails every database that
has ever deleted anything, which is the steady state of the free-list design.
The collision that does matter — a page the follower's own free-list chain or
commit-history tree still hosts — is invisible from the producer. The follower
keeps both across an apply, and the base commit's recorded free-list root is
superseded writer metadata whose pages a later commit may already have recycled,
so reading it to guess would authenticate a page that is no longer a free-list
page at all. The follower owns that check and runs it before any byte reaches
main.db, reportingPagedbError::SnapshotBasePageReusedwith the offendingpage id.
That error is deliberately not a corruption variant. Both states are internally
sound; they simply cannot be related by a page delta, and the remedy is a full
snapshot or a nearer base commit.
Full snapshot export and restore
Export fails if the source cannot supply every main page or segment file the
authenticated state requires, synchronizes copied files before publishing the
manifest, and records the exact manifest length so trailing data cannot be
silently accepted.
Restore requires an empty destination, so a snapshot is never merged into
unrelated pre-existing pages or segment files. Segment filenames are screened
during the copy and counted against the manifest before anything opens the
destination, so an undeclared sidecar is reported as artifact corruption rather
than as whatever error open-time recovery happens to raise.
Before the restored database is exposed, PageDB authenticates the restored
header, active root, reachable tree pages, segment manifests, and segment data
pages, and verifies the active segment set is exactly the set the manifest
declares: missing, renamed, corrupt, and extra undeclared files all fail.
Cleanup after a failure is scoped by recorded ownership rather than inferred
from an
io::ErrorKind. A destination the operation created is removed; adestination the caller pre-created is emptied but left in place.
Incremental export and application
Incremental export rejects a missing base commit and fails if any changed main
page or newly required segment file cannot be copied.
Incremental apply performs a complete validation pass before publication. It
requires follower mode and a matching realm; the follower's authenticated commit
to equal the declared base commit; a strictly advancing target commit; compatible
manifest fields; no trailing manifest bytes; complete delta records with no
truncation; one record per page ID; page IDs inside the target allocation bound
and never a reserved page; every target-reachable changed page to be present;
and every declared new segment to exist and authenticate as the expected
identity.
Base page sets and the base segment set are drawn from a single sample of the
published snapshot, so a concurrent
gc_now— which Follower mode permits —cannot split the base state across the comparison.
Pager cache coherency on retry
Raw incremental writes bypass the normal pager write path, so a validation read
during a failed apply can leave target-page plaintext in the pager cache. A
regression test forces a late failure through a corrupt segment, then corrupts a
target-root page and retries; before the fix the retry incorrectly succeeded
because validation reused the cached page rather than authenticating the bytes
now on disk.
Apply now resets cached main pages immediately after the raw delta writes and
before authenticating the target tree. The placement is exact: earlier would not
cover the raw writes, later would let stale plaintext influence validation.
Segment publication, tombstones, and recovery
The apply path stages and authenticates every promoted segment before
publication, computes the segment set reachable from the target catalog, and
emits journal actions for both promotion and tombstoning — so a target that
removes a segment still present in the follower directory does not leave the
physical store holding state the target catalog does not represent. Replay can
finish the same operation after an interruption.
Also covered: segment sets larger than one journal page, directory-sync failures,
deferred journal cleanup while readers remain pinned, replay after a promotion
failure poisons the handle, and serialization of concurrent applies before any
raw page write.
Regression coverage
The snapshot suite is the deliverable as much as the source change. Highlights:
incremental_round_trip_survives_page_reuse_below_the_base_cursor— a fulldelete/refill/export/apply cycle, asserting that reuse actually occurred so it
cannot silently degrade into the append-only case.
incremental_snapshot_exports_reused_pages_below_base_cursor— parsespages.deltato prove at least one shipped record sits below the base cursor.apply_incremental_refuses_to_overwrite_a_base_live_page_without_mutating_base— injects the base snapshot's own active root and asserts
SnapshotBasePageReusednames it, and that the base survives the refusal.Two historical expectations were ruled out rather than transplanted:
derived from authenticated follower metadata, and the cursor is not treated as
a liveness boundary at all;
authenticated and part of the published base state, so clearing it would now
be incorrect.
Verification
On the merged head:
cargo nextest run -p pagedb --all-features --no-fail-fast: 492 passed,5 skipped;
PAGEDB_INVARIANT_CHECKS=1: 492 passed, 5 skipped;cargo fmt --all --check: pass;cargo clippy -p pagedb --all-targets --all-features -- -D warnings: pass;cargo doc -p pagedb --no-deps --all-features: no new warnings;The page-reuse regression was confirmed to fail against the pre-fix source and
pass after it.
Review focus
whether any other follower-local page set belongs in the write guard;
across every interruption point;
restore and incremental apply;
a durability error.
The change favors rejection over partial recovery whenever an artifact cannot
prove the exact authenticated target state.